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1.  Introduction 

Let  (X^),  t  6  T  ■  [0,“),  be  an  Irreducible  Markov  chain  In  contin¬ 
uous  time  with  state  space  I  =  (o,l,2,...).  The  stationary  transition 
probability  matrix  P(t)  »>(p^j(t))  Is  assumed  to  be  measurable  and 
satisfy 


(1.1) 


Plj(t)  >  0,  I] 

J 

P(t+s)  =  P(t)  P(s),  P(0+)  =  I  , 


1>  J  e  I  , 


for  all  t,  8  €  T.  In  addition,  the  states  are  assumed  to  be  stable; 


Pii(t)-i 


0  >  p^^'(o)  =  1^  , 


(1.2) 


t|0 


q.  >  -  00  ,  lei  , 


0  <  Pi/(0)  =  11m  + 

t|0 


=  qij  <  +  c»  , 


1  J  e  I  . 


The  matrix  Q  =  (q^j)  called  the  Q-matrlx  or  Infinitesimal  generator 

matrix  of  the  process,  and  It  Is  assumed  to  be  conservative,  l.e., 

I]  q^.  =  0,  1  6  I.  For  simplicity,  this  type  of  Markov  chain  will  be 
J  ^ 

referred  to  as  a  simple  continuous  time  Markov  chain  (SdC).  A  thorough 
treatise  on  the  properties  of  a  SCMC  Is  contained  In  [1]. 

In  [8]^  [9]  the  solutions  to  the  equations  y  Q  =  0  were  Inves¬ 
tigated.  niese  statlonarlty  equations  are  obtained  by  setting  the 


derivatives  equal  to  zero  in  the  forward  Kolmogorov  equations  P'(t)  - 
F(t)Q,  and  are  the  continuous  time  analog  of  the  statlonarity  equations 
xP  >  X  for  a  discrete  time  Markov  chain  (with  stationary  one-step 
transition  probahllity  matrix  P  >  (p^^)).  In  particular,  it  was  shown 
that,  under  the  minimality  assuiq;>tlon  described  below,  a  NSC  for  the 
SCKK7  to  be  positive  recurrent  is  for  the  equations  yQ  -  0  to  have  a 
convergent,  positive  solution  y  =  (yQiy^^/yg# • • • ) •  The  solution  is 
unique  (up  to  a  multiplicative  constant): 

(1.3)  ^1  ”  *1  “  ^11^^^  >  i  €  I  . 

t  -+  «0 

In  [9]  yQ  0  was  also  shown  to  have  a  tuilque,  positive  solution  in  a 
null  recurrent  chain,  and  for  any  recurrent  chain  (positive  or  null)  the 
relationship  between  the  unique  stationary  measiures  of  the  SCMC  and  its 
imbedded  discrete  time  Markov  chain  was  obtained. 

The  analogous  statlonarity  theorem  for  positive  recurrent  chains 
in  discrete  time  is  due  to  Foster  [4]  with  an  earlier,  less  general 
version  being  given  by  Feller  (3),  P*  325  (see  also  Chung  [1],  p.  33). 
Poster  also  gives  three  additional  theorems  on  conditions  for  recurrence, 
ergodlclty,  etc.  in  a  discrete  time  chain.  The  purpose  of  this  paper 
is  to  extend  these  additional  theorems  to  continuous  time. 

The  minimality  assumption  referred  to  above  which  is  necessary  for 
the  validity  of  the  preceding  theorems  in  continuous  time  is: 

Minimality  Assumption;  The  SCMC  is  uniquely  defined  by  its  Q-matrlx; 
l.e.,  the  minimal  process  is  an  honest  process  (see  [1]  for  details). 

It  will  be  necessary  to  impose  this  assumption  in  Theorem  2  below  but 
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will  not  be  needed  in  Ibeorem  1.  It  excludes  from  consideration  those 
processes  which  can  explode  to  +  <»  in  finite  time.  Various  necessary 
and  sufficient  conditions  on  the  Q-matrlx  for  the  alnlmallty  assunptlon 
to  hold  have  been  derived  and  can  be  found  elsewhere.  For  reference, 
see  Chung  [1]  and  Reuter  [10]. 


2.  Results 

In  the  proofs  of  this  section  it  will  be  necessary  to  refer  to  the 
Imbedded  chain  of  the  SCMC.  This  is  the  Irreducible  (but  possibly 
periodic)  discrete  time  Markov  chain  n  =  0,1,2,  •••,  with 

stationary  transition  probability  matrix  P  =>  (Pj^j)  where 


^IJ  “  '^l/*^l  * 


1  /  J  6  I 


(2.1) 


Pji  =  0  ,  i  6  I  . 

Ibe  Imbedded  Markov  chain  (X^^)  simply  records  the  sequence  of  states 
through  which  the  SCMC  passes  without  regard  to  the  amount  of  time  re¬ 
quired  for  the  transitions. 


Theorem  1;  (a)  The  SCMC  {X^}  is  recurrent  if  there  exists  a  sequence 

z  =  such  that  (i)  z^ -»  +  »  as  n -»  +  «  and  (li) 

Qz  <  0  except  for  the  first  coordinate. 

(b)  A  NSC  for  the  SCMC  (X^)  to  be  non- recurrent  is  that 
there  exist  a  bounded  non-constant  sequence  z  =:  (z^,z^,Z2, . • .)  such 
that  Qz  >  0  except  for  the  first  coordinate. 


Proof;  The  proofs  of  (a)  and  (b)  are  immediate  and  can  be  given  together. 
Ibe  system  of  Ineque^llties  or  equalities  Qz  <  0,  >0  can  be  written  as 
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(2.2) 


i  /  0  e  I 


r,.  »ij'j  i  * 
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Division  by  yields 

OQ 

(2.5)  E  PljZ,  <  .  =  .  1  /  O  €  I 

where  P  *  (Pjj)  transition  matrix  of  the  imbedded  chain.  Under 

the  conditions  on  z  in  (a)  the  system  of  inequalities  (2.5)  i]i3>lies 
recurrence  for  the  Imbedded  chain  by  Theorem  ^  of  [4].  Similarly,  under 
the  conditions  on  z  in  (b)  the  system  of  equations  (2.5)  is  a  NSC  for 
the  transience  of  the  imbedded  chain  by  ^nieorem  4  of  [4].  But  the 
recurrence  or  transience  of  the  imbedded  chain  is  identical  to  the  re¬ 
currence  or  non- recurrence, respectively,  of  the  SCMC.  Recurrence  is 
Independent  of  the  time  component.  | | 

The  term  "non-recurrent"  rather  than  "transient"  is  used  here  in 
dealing  with  a  SCMC  because  of  the  two  possible  types  of  path  function 
behavior.  A  SCMC  can  be  non-explosive  (l.e.,  satisfy  the  minimality 
assumption)  but  have  transient  states  In  the  sense  that  a  return  to 
each  ha?  probability  less  them  one,  or  it  can  be  explosive  and  reach 
+  «  In  finite  time  with  positive  probability.  In  both  cases  the  states 
of  the  Imbedded  chain  are  transient. 

Note  that  it  Is  not  necessary  to  impose  the  minimality  assumption 
in  Theorem  1.  Hie  fact  that  the  imbedded  chain  has  not  been  defined 
beyond  the  first  Infinity  does  not  cause  any  difficulty.  Should  Qz  =  0 
not  have  a  bounded  non-constant  solution  or  Qz  <  0  have  a  solution 
whose  coordinates  tend  to  <»,  the  imbedded  chain  is  recurrent,  and 


by  necessity  the  8CMC  is  uniquely  defined.  Should  Q?  -  0  have  a 
bounded  non-constant  solution^  the  Imbedded  chain  is  transient;  the  SCtC 
is  then  either  ei^losive  or  non-explosive  and  transient. 

This  theorem,  particularly  part  (b),  is  motivated  by  the  following 
consideration.  Let  f^^  be  the  probability  that  if  the  SdC  (or  its 
Imbedded  chain)  starts  in  state  1,  it  reaches  state  0  eventually. 

If  f^Q  la  defined  to  be  1,  then  the  f satisfy  the  equations 

(2.M 

In  a  recurrent  chain  f^^  s  i,  but  in  a  transient  chain  f 1,  so 
the  f^Q  constitute  a  bounded,  non-constant  solution  to  (2.4). 

In  an  earlier  paper  [6]  Khrlin  and  McGregor  extended  Foster's 
nieorems  4  and  5  to  birth  and  death  processes.  Utilizing  the  special 
structure  of  these  processes  they  also  established  necessity  in  part 
(a),  ^is  does  not  seem  to  be  true  in  general  (see  [4]). 

Theorem  2:  under  the  minimality  assusiptlon  a  NSC  that  the  SCMC  be  positive 
recurrent  is  that  the  inequalities 

(2.5)  f  <  -  .  ,  M  0  .  I 

(l.e.,  Qz  <  -  1  except  for  the  first  coordinate)  have  a  non-negative 
solution  z  which  satisfies 

QO 

(2.6) 

(i.e.,  |(Qs)q|  <  +  «). 
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Proof;  (NOcesalty)  Let  be  the  expected  time  It  takes  the  SCMC 

to  reach  state  0  fron  state  1  /  0  e  I>  m„  -  0*  For  a  positive 
recurrent  SCMC  <  •«■  «.  (These  expected  flrst>passage  times  satisfy 
the  equations 


(2.7) 


m 


10 


lj”jO  ' 


i  /O  e  I  , 


where  the  first  term  on  the  right  is  the  expected  length  of  time  spent 
In  state  1  and  the  second  term  is  the  expected  time  to  reach  0  after 
the  process  leaves  state  1.  Multiplication  of  (2.7)  by  q^^  and 
rearrangement  of  terms  yields  (2.5)  with  equality  for  z.  =  m.  . 

J  JU 

since  the  chain  Is  positive  recurrent,  the  mean  recurrence  time  to 
state  0  is  finite)  l.e., 


(2.8) 


_1 


which  implies  (2.6). 

(Sufficiency)  Rearrangement  of  terms  in  (2.5)  gives 


(2.9) 


00 


I  P 
J=o 


-  *1 
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1  /  0  €  I 


Without  loss  of  generality  assume  z^  =  0.  Consider  the  iterative 
inequality  obtained  by  applying 


J=0 


V 


k-0 


3  (“-^) 

•ik 


J*0 


6 


(2.10) 


.  (n-1) 


<*k-^) 


+  P 


(n-l) 
10 


J-0 


^0J*J 


^Ik  ’ 


.  y  p  (n-l)  J.  +  p 

^Ik  a.  ^in 


k-o 


10 


X) 


where  X  -  XI  Pni*i  ^  "  by  (2.6). 

j-0  *’ 

(Qie  (n-l) -fold  Interatlon  of  this  Inequality  produces  the  following 
Inequality: 


(2.11)  0<  £  p 


J-0 


(■»)* 

IJ  i 


Il-X 

-  E  Z 

vsl  k>0 


(v)  _1 


''Ik 


1  n.*X  f  V 

^  't  ^  ’ 


00  00  «  a 

The  series  Y  Y  p.5^V<lv  1®  divergent  by  the  minimality  aesianptlon 
v-1  k^  “  * 

since  It  Is  the  expected  time  required  to  make  an  Infinite  number  of 
transitions  after  leaving  state  1.  For  a  rigorous  proof  see  inieorem 
II.  19.1,  Corollary  1,  of  [1].  But  this  means  that  ^  must  be 


v=l 


divergent  In  order  to  preserve  the  non-negatlvlty  In  (2.11).  Hence, 
the  SCMC  Is  recurrent. 

To  establish  positive  recurrence  sum  the  Inequality  (2.10)  for 
n  -  2 , . . . , N+1 . 

(2.12)  I  I  p  <“>.  <ri  -  r  I  p  (■')  ^ 

n-2  J-0  *'  n^  k-0  n^  k-0  “  ^k 
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.  (i-  +  1)  7  p  (x) 

«0  n^a 


Rearrangement  and  cancellation  produces 


ctb  nti  ^  ktb  ktb  ^  %  n^?l 


k-0 

(2.13) 


5  .1  ^  Ip 


k=0 


n=l 


(n) 

10 


Divide  both  sides  of  (2.13)  by  £  (which  is  positive  for  N 


n=l 


Ih 


sufficiently  large)  for  any  he  I.  As  N  -»»  Fatou's  lemma  gives 


(2.14)  I 
k=0 


lim 


Ip 

n=l 


(n) 

ik 


^  ^10 

1  ^  \  \  -.4  n=l  ^ 

—  <  (“  +  1)  lim 


(n) 


V  T,  (n)  /  '  N  -»» 

A.  Pih 


n=l 


2_Pih 


(n) 


n=l 


the  right  hand  side  reducing  to  a  single  term  since 


I  ht^  ■  * 


n=l 


in  a  recurrent  chain.  These  limits  exist  by  the  Doeblin  ratio  limit 
theorem  and  have  been  evaluated  by  Chung.  For  a  recurrent  chain 

N 


(2.15) 


lim 
N  -K» 


Ip 

n“l 


(n) 

Ik 


l^ik 


n®! 


■ih 


for  any  i  e  I,  where  expected  number  of  visits  to  state 

k  between  visits  to  state  ‘*(|P*j/  *  ^)*  (^®^  reference  see  (1], 

Sec.  1.9).  From  (2.lU)  and  (2.15) 


(2.16) 


k-0  *  ^k  "  '*0 
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Deman  [2]  shoved  that  tor  a  recurrent  chain  the  i  0^1,2,..., 

constitute  the  unique  (except  for  a  multiplicative  constant)  positive 
solution  of  the  equations  xP  «  x.  In  a  recurrent  chain  the  unique 
solutions  of  yQ  «  0  and  xP  *  x  are  related  by  y^^  «  *l/*^i 
Theorem  3  of  [9])-  Thus,  by  (2.16)  y^  *  ®  positive, 

convergent  solution  to  yQ  =  0  so  the  SCMC  is  positive  recurrent  (by 
Theorem  1  of  [9])-  || 

The  motivation  for  this  theorem  is  clearly  contained  in  the  necessity 
part  of  the  proof  where  (2.5)  holds  with  equality  for  Zj  =  mj^.  That 
the  equalities  can  be  replaced  with  inequalities  in  the  sufficiency 
condition  is  a  trivial  bonus  of  the  proof. 

The  minimality  assumption  is  essential  for  the  validity  of  the 
sufficiency  part  of  the  theorem.  For  a  counter-example  without  it  take 
a  birth  and  death  process  with 


(2.17) 


=  0 


n*0 


<  + 


<  +  00 


where  ’’  ^n'  “  l>2,...,p^  =  1.  Such  a  birth 

and  death  process  is  explosive  (see  [5])*  However, 


n  =  1,2, . . ., 


satisfies  the  equations  T  ^  ’’  ^>2,...,  for  any  z^ . 

»  ^  ^ 

(  Y  q^.z.  <  +  00  holds  trivially.)  For  sufficiently  large  z 
J*0  ^  “ 

will  be  positive  for  all  n  since  the  negative  series  in  (2.l6)  is 


convergent. 
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Kingman  [7]  proved  this  theorem  for  bounded  by  a  different 
method.  Since  boundedness  of  the  q^  guaremtees  the  minimality 
assumption  but  is  not  necessary  for  it  to  hold.  Theorem  2  vould  con¬ 
stitute  an  extension  of  Kingnan's  result.  An  application  of  the  suf¬ 
ficiency  condition  to  parallel  queues  can  also  be  found  in  [?]. 
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